Skip to content

Python: Include Bing Grounding Tool call results in invoke_stream responses #12265

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
May 27, 2025

Conversation

KanchiShimono
Copy link
Contributor

Motivation and Context

According to the Terms of Service for Grounding with Bing Search, the tool’s call results must be displayed in their original format.
Currently, in the AzureAIAgent invoke/invoke_stream responses, search results are included in the AnnotationContent, but the search query itself is not.
This PR updates the response of invoke/invoke_stream to include the Bing Grounding Tool’s call results, so that the Bing search query can also be displayed.

Description

  • Fixe an issue where errors occurred when handling Bing Grounding Tool results in streaming mode
  • Update invoke/invoke_stream responses to include the Bing Grounding Tool call results

Contribution Checklist

@KanchiShimono KanchiShimono requested a review from a team as a code owner May 25, 2025 07:57
@markwallace-microsoft markwallace-microsoft added the python Pull requests for the Python Semantic Kernel label May 25, 2025
@github-actions github-actions bot changed the title Include Bing Grounding Tool call results in invoke/invoke_stream responses Python: Include Bing Grounding Tool call results in invoke/invoke_stream responses May 25, 2025
@moonbox3
Copy link
Contributor

Thanks for this, @KanchiShimono. This is using the azure-ai-agents>=1.0.0 and azure-ai-projects>=1.0.0b11 packages, right? Looks like they added support to get this info, because in the current way to handle the tool, we didn't have this info.

Copy link
Contributor Author

@KanchiShimono KanchiShimono left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@moonbox3
Thank you for your review.

As you mentioned, I'm using the azure-ai-agents==1.0.0 and azure-ai-projects==1.0.0b11 packages.

By the way, for content that needs special handling, such as Bing search queries, I think it would be ideal to have a dedicated content type like BingGroundingCallContent.
With FunctionCallContent, we have to check the function_name, which can make handling the results a bit more complicated.

@moonbox3
Copy link
Contributor

@moonbox3 Thank you for your review.

As you mentioned, I'm using the azure-ai-agents==1.0.0 and azure-ai-projects==1.0.0b11 packages.

By the way, for content that needs special handling, such as Bing search queries, I think it would be ideal to have a dedicated content type like BingGroundingCallContent. With FunctionCallContent, we have to check the function_name, which can make handling the results a bit more complicated.

Thanks for the comment, @KanchiShimono. I understand that it would be ideal to have a more specific BingGroundingCallContent but that's tricky for the SK SDK, where we aim to have content types that have representation across different Agents/AI Services/Providers, etc. The BingGroundingCallContent has such a slim touch-point (Azure AI Agent), that it's difficult to justify introducing a brand new content type just for that. Could you help me understand, please, what is so tricky about using FunctionCallContent with the Bing Grounding Tool? Thank you.

@KanchiShimono
Copy link
Contributor Author

@moonbox3
Thank you for your comment.

I agree with your point that adding a highly specific type like BingGroundingCallContent can be difficult. I want to clarify that I’m not strongly requesting the addition of BingGroundingCallContent right now,it was just a casual suggestion.

Here’s what I find tricky about using FunctionCallContent in this context:

  • To know that the function_name in FunctionCallContent should be bing_grounding, we have to look into the SK source code, it’s not obvious from the outside.
  • If the Bing Grounding Tool result returned by azure-ai-agents (RunStepDeltaToolCall.type) ever changes from bing_grounding, string-based comparisons could easily miss that change until runtime, making it harder and riskier to upgrade package versions.
  • With a dedicated type like BingGroundingCallContent, type checking could absorb such changes within the SK SDK itself. This would let users keep their code unchanged in many cases. Also, changes or removals of the type can be detected before runtime using static type checkers (like mypy or Pylance), which reduces the risk of breakage.

@moonbox3
Copy link
Contributor

Hi @KanchiShimono, thanks for the additional comments, they’re very helpful. I agree with your points, and we’ll discuss your suggestion about more scoped content types internally with the SK team.

Now that the agent service is GA, we don’t expect any further breaking changes (for example, bing_grounding being renamed). For type safety, we should be able to type check against the Azure SDK’s BingGroundingToolCall (or whatever the final type is named), to make sure that we’re handling the correct tool call at that layer.

However, once tool calls are converted to SK’s FunctionCallContent, things become less strict since plugins and functions can use arbitrary names. Given this flexibility, the most practical approach for now is to make sure the Bing Tool Call type is well documented, so developers understand how the tool’s function_name will appear within SK’s function call content.

Let me know if there’s anything I missed or if you have further suggestions.

@KanchiShimono
Copy link
Contributor Author

@moonbox3
Thank you for your comments.
Your explanation helped me better understand some practical approaches to handling Bing Grounding-like content with SK.
I really appreciate how well-organized the SK documentation and samples are, and I think adding this information there would be extremely helpful for developers.

I don’t have any further comments at this time.
Thank you for sincerely listening to and engaging with user feedback.

@moonbox3
Copy link
Contributor

Thanks @KanchiShimono. I will make a new issue to track adding the updated Bing Tool content to the AzureAIAgent page in our learn site, once this change goes in. Appreciate the help with this.

@markwallace-microsoft
Copy link
Member

markwallace-microsoft commented May 27, 2025

Python Test Coverage

Python Test Coverage Report •
FileStmtsMissCoverMissing
agents/azure_ai
   agent_content_generation.py2105076%72–73, 75, 93, 95, 217, 220, 260, 285–286, 294, 302, 304, 312, 319, 355, 394–395, 397, 399–402, 412, 425–426, 428, 430–434, 443, 451, 477–484, 490–493, 498, 503–504, 511
TOTAL26990456783% 

Python Unit Test Overview

Tests Skipped Failures Errors Time
3616 18 💤 0 ❌ 0 🔥 1m 52s ⏱️

@moonbox3 moonbox3 changed the title Python: Include Bing Grounding Tool call results in invoke/invoke_stream responses Python: Include Bing Grounding Tool call results in invoke_stream responses May 27, 2025
@moonbox3 moonbox3 enabled auto-merge May 27, 2025 04:37
@moonbox3 moonbox3 added this pull request to the merge queue May 27, 2025
Merged via the queue into microsoft:main with commit 7295458 May 27, 2025
28 checks passed
@KanchiShimono KanchiShimono deleted the bing-grounding-tool-query branch June 13, 2025 02:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
python Pull requests for the Python Semantic Kernel
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants